<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
    <meta charset="utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" /><link rel="canonical" href="https://example.com/transform.html" />
      <link rel="shortcut icon" href="img/favicon.ico" />
    <title>Transform: Convert data - ten's documentation</title>
    <link rel="stylesheet" href="css/theme.css" />
    <link rel="stylesheet" href="css/theme_extra.css" />
        <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/10.5.0/styles/github.min.css" />
    
      <script>
        // Current page data
        var mkdocs_page_name = "Transform: Convert data";
        var mkdocs_page_input_path = "transform.md";
        var mkdocs_page_url = "/transform.html";
      </script>
    
    <script src="js/jquery-3.6.0.min.js" defer></script>
    <!--[if lt IE 9]>
      <script src="js/html5shiv.min.js"></script>
    <![endif]-->
      <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/10.5.0/highlight.min.js"></script>
      <script>hljs.initHighlightingOnLoad();</script> 
</head>

<body class="wy-body-for-nav" role="document">

  <div class="wy-grid-for-nav">
    <nav data-toggle="wy-nav-shift" class="wy-nav-side stickynav">
    <div class="wy-side-scroll">
      <div class="wy-side-nav-search">
          <a href="index.html" class="icon icon-home"> ten's documentation
        </a><div role="search">
  <form id ="rtd-search-form" class="wy-form" action="./search.html" method="get">
      <input type="text" name="q" placeholder="Search docs" title="Type search term here" />
  </form>
</div>
      </div>

      <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
              <ul>
                <li class="toctree-l1"><a class="reference internal" href="index.html">Home</a>
                </li>
              </ul>
              <ul>
                <li class="toctree-l1"><a class="reference internal" href="flow.html">Flow: Input, output, and flow control</a>
                </li>
              </ul>
              <ul class="current">
                <li class="toctree-l1 current"><a class="reference internal current" href="transform.html">Transform: Convert data</a>
    <ul class="current">
    <li class="toctree-l2"><a class="reference internal" href="#json">JSON</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#base-64-encoding-base64">Base 64 encoding (base64)</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#query-string-qs">Query string (qs)</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#table">Table</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#hashing">Hashing</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#others">Others</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#multiform">Multiform</a>
    </li>
    <li class="toctree-l2"><a class="reference internal" href="#cli-the-tf-program">CLI: the tf program</a>
    </li>
    </ul>
                </li>
              </ul>
              <ul>
                <li class="toctree-l1"><a class="reference internal" href="http.html">HTTP: requests on steroids</a>
                </li>
              </ul>
              <ul>
                <li class="toctree-l1"><a class="reference internal" href="shell.html">Shell: run commands</a>
                </li>
              </ul>
              <ul>
                <li class="toctree-l1"><a class="reference internal" href="fs.html">Filesystem: manage folders and files</a>
                </li>
              </ul>
              <ul>
                <li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorials and examples</a>
                </li>
              </ul>
      </div>
    </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
      <nav class="wy-nav-top" role="navigation" aria-label="Mobile navigation menu">
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="index.html">ten's documentation</a>
        
      </nav>
      <div class="wy-nav-content">
        <div class="rst-content"><div role="navigation" aria-label="breadcrumbs navigation">
  <ul class="wy-breadcrumbs">
    <li><a href="index.html" class="icon icon-home" alt="Docs"></a> &raquo;</li>
      <li>Transform: Convert data</li>
    <li class="wy-breadcrumbs-aside">
    </li>
  </ul>
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
            <div class="section" itemprop="articleBody">
              
                <h1 id="transform-convert-data">Transform: Convert data</h1>
<p>Exploits need to manipulate data easily: encode it into base64, compute its hash, HTML-decode it, <em>etc.</em> That's what <code>transform</code> is for.</p>
<p>Each category of transformation is available as a submodule of <code>transform</code>.
When you do  <code>from ten import *</code>, <code>transform</code> is imported as <code>tf</code>, and the most used modules, such as <code>json</code> or <code>base64</code>, are included as root modules.</p>
<p>Each category of transformation is available as a submodule of <code>transform</code>.</p>
<pre><code class="language-python">&gt;&gt;&gt; from tenlib.transform import base64, html
&gt;&gt;&gt; base64.decode('dGVzdA==')
b'test'
&gt;&gt;&gt; html.decode('&amp;lt;div&amp;gt;test&amp;lt;/div&amp;gt;')
'&lt;div&gt;test&lt;/div&gt;'
</code></pre>
<p>Methods support both <code>bytes</code> and <code>str</code> as input, as well as lists and dicts. Refer to <a href="#multiform">Multiform</a> for more details.</p>
<p>Here are a few examples for the most common transformations.</p>
<h2 id="json">JSON</h2>
<table>
<tr><th>python</th><th>output</th></tr>

<tr>
<td>


<pre><code class="language-python">json.encode({&quot;a&quot;: &quot;b&quot;, &quot;c&quot;: &quot;d&quot;})
</code></pre>


</td>
<td>


<pre><code class="language-python">'{&quot;a&quot;:&quot;b&quot;,&quot;c&quot;:&quot;d&quot;}'
</code></pre>


</td>
</tr>
<tr>
<td>


<pre><code class="language-python">json.decode('{&quot;a&quot;: &quot;b&quot;, &quot;c&quot;: &quot;d&quot;}')
</code></pre>


</td>
<td>


<pre><code class="language-python">{&quot;a&quot;: &quot;b&quot;, &quot;c&quot;: &quot;d&quot;}
</code></pre>


</td>
</tr>

</table>

<h2 id="base-64-encoding-base64">Base 64 encoding (base64)</h2>
<table>
<tr><th>python</th><th>output</th></tr>

<tr>
<td>


<pre><code class="language-python">base64.encode('test')
</code></pre>


</td>
<td>


<pre><code class="language-python">'dGVzdA=='
</code></pre>


</td>
</tr>
<tr>
<td>


<pre><code class="language-python">base64.decode('dGVzdA==')
</code></pre>


</td>
<td>


<pre><code class="language-python">b'test'
</code></pre>


</td>
</tr>

</table>

<h2 id="query-string-qs">Query string (qs)</h2>
<p>Parse query string and URL-encode/decode.</p>
<table>
<tr><th>python</th><th>output</th></tr>

<tr>
<td>


<pre><code class="language-python">qs.parse('k1=v1&amp;k2=v2')
</code></pre>


</td>
<td>


<pre><code class="language-python">{'k1': 'v1', 'k2': 'v2'}
</code></pre>


</td>
</tr>

<tr>
<td>


<pre><code class="language-python">qs.unparse({'k1': 'v1', 'k2': 'v2'})
</code></pre>


</td>
<td>


<pre><code class="language-python">'k1=v1&amp;k2=v2'
</code></pre>


</td>
</tr>

<tr>
<td>


<pre><code class="language-python">qs.decode('%41%42%43%3a%44%45%46')
</code></pre>


</td>
<td>


<pre><code class="language-python">'ABC:DEF'
</code></pre>


</td>
</tr>
<tr>
<td>


<pre><code class="language-python">qs.encode('ABC:DEF')
</code></pre>


</td>
<td>


<pre><code class="language-python">'ABC%3aDEF'
</code></pre>


</td>
</tr>
<tr>
<td>


<pre><code class="language-python">qs.encode_all('ABC:DEF')
</code></pre>


</td>
<td>


<pre><code class="language-python">'%41%42%43%3a%44%45%46'
</code></pre>


</td>
</tr>

</table>

<h2 id="table">Table</h2>
<p>Convert data formatted into a table (such as CSV) to a list (of list), and back.
This example converts an array of colon separated lines into a table.</p>
<table>
<tr><th>python</th><th>output</th></tr>

<tr>
<td>


<pre><code class="language-python">table.split('''\
username:password:email
admin:dUD6s55:admin@site.com
moderator:123456!:moderator@site.com
user:Password1:user@gmail.com\
''', '\n', ':')
</code></pre>


</td>
<td>


<pre><code class="language-python">[
    [b'username', b'password', b'email'],
    [b'admin', b'dUD6s55', b'admin@site.com'],
    [b'moderator', b'123456!', b'moderator@site.com'],
    [b'user', b'Password1', b'user@gmail.com'],
]
</code></pre>


</td>
</tr>
<tr>
<td>


<pre><code class="language-python">table.join([
    [b'username', b'password', b'email'],
    [b'admin', b'dUD6s55', b'admin@site.com'],
    [b'moderator', b'123456!', b'moderator@site.com'],
    [b'user', b'Password1', b'user@gmail.com'],
], '\n', ':')
</code></pre>


</td>
<td>


<pre><code class="language-python">'''\
username:password:email
admin:dUD6s55:admin@site.com
moderator:123456!:moderator@site.com
user:Password1:user@gmail.com\
'''
</code></pre>


</td>
</tr>

</table>

<h2 id="hashing">Hashing</h2>
<table>
<tr><th>python</th><th>output</th></tr>

<tr>
<td>


<pre><code class="language-python">hashing.md5('test')
</code></pre>


</td>
<td>


<pre><code class="language-python">'098f6bcd4621d373cade4e832627b4f6'
</code></pre>


</td>
</tr>
<tr>
<td>


<pre><code class="language-python">hashing.sha1('test')
</code></pre>


</td>
<td>


<pre><code class="language-python">'a94a8fe5ccb19ba61c4c0873d391e987982fbbd3'
</code></pre>


</td>
</tr>

</table>

<h2 id="others">Others</h2>
<p>Lots of other transforms are available. Refer to the <a href="../tenlib/transform/index.html">documentation</a> for more details.</p>
<h2 id="multiform">Multiform</h2>
<p>As you may have seen, you can feed either bytes or strings to the conversion
functions, and they will handle the convertion into the proper type. This
behaviour also works with <code>dict</code>s and <code>list</code>s:</p>
<table>
<tr><th>python</th><th>output</th></tr>

<tr>
<td>


<pre><code class="language-python">base64.encode('test')
</code></pre>


</td>
<td>


<pre><code class="language-python">'dGVzdA=='
</code></pre>


</td>
</tr>

<tr>
<td>


<pre><code class="language-python">base64.encode(b'test')
</code></pre>


</td>
<td>


<pre><code class="language-python">'dGVzdA=='
</code></pre>


</td>
</tr>

<tr>
<td>


<pre><code class="language-python">base64.encode([
    'test',
    b'test2',
    'test3'
])
</code></pre>


</td>
<td>


<pre><code class="language-python">[
    'dGVzdA==',
    'dGVzdDI=',
    'dGVzdDM='
]
</code></pre>


</td>
</tr>
<tr>
<td>


<pre><code class="language-python">qs.decode({
    'k1': '%41%42%43',
    'k2': '%44%45%46'
})
</code></pre>


</td>
<td>


<pre><code class="language-python">{
    'k1': 'ABC',
    'k2': 'DEF'
}
</code></pre>


</td>
</tr>

</table>

<h2 id="cli-the-tf-program">CLI: the <code>tf</code> program</h2>
<p>The <code>tf</code> program makes transforms available from the CLI as well:</p>
<pre><code class="language-shell">$ echo 'dGVzdA==' | tf base64.decode
test
</code></pre>
<p>The filters can be chained. Here, we convert JSON into an URL-encoded string:</p>
<pre><code class="language-shell">$ echo '{&quot;a&quot;:&quot;b&quot;,&quot;c&quot;:&quot;d&quot;}' | tf json.decode qs.unparse
a=b&amp;c=d
</code></pre>
<p>Check the <code>--help</code> for further details.</p>
              
            </div>
          </div><footer>
    <div class="rst-footer-buttons" role="navigation" aria-label="Footer Navigation">
        <a href="flow.html" class="btn btn-neutral float-left" title="Flow: Input, output, and flow control"><span class="icon icon-circle-arrow-left"></span> Previous</a>
        <a href="http.html" class="btn btn-neutral float-right" title="HTTP: requests on steroids">Next <span class="icon icon-circle-arrow-right"></span></a>
    </div>

  <hr/>

  <div role="contentinfo">
    <!-- Copyright etc -->
  </div>

  Built with <a href="https://www.mkdocs.org/">MkDocs</a> using a <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
          
        </div>
      </div>

    </section>

  </div>

  <div class="rst-versions" role="note" aria-label="Versions">
  <span class="rst-current-version" data-toggle="rst-current-version">
    
    
      <span><a href="flow.html" style="color: #fcfcfc">&laquo; Previous</a></span>
    
    
      <span><a href="http.html" style="color: #fcfcfc">Next &raquo;</a></span>
    
  </span>
</div>
    <script>var base_url = '.';</script>
    <script src="js/theme_extra.js" defer></script>
    <script src="js/theme.js" defer></script>
      <script src="search/main.js" defer></script>
    <script defer>
        window.onload = function () {
            SphinxRtdTheme.Navigation.enable(true);
        };
    </script>

</body>
</html>
